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1. INTRODUCTION 

Enterprise Architecture (EA) provides a comprehensive strategy and environment for aligning 
enterprise business and Information Technology (IT) [1, 2]. In order to provide an appropriate environment 
for alignment between business and IT, EA describes the current architecture (As-Is), elaborates the desired 
architecture (To-Be), and represents the migration plan for transition from current architecture to desired 
architecture for the enterprise [3, 4]. Four architectural levels business, data, application, and infrastructure 
need to be described in these three aforementioned stages of an EA project [5-7]. 

EA project comprises a set of developed EA artefacts including the related information systems, 
documents and enterprise’s roadmaps in accordance with the defined EA project objectives. In othis regards, 
the integration among and between developed information systems are important [8]. This integration 
provides the appropriate environment for better connectivity of enterprise’s business [9, 10]. 

EA evaluation is defined as the process of determining the merit, worth, and value of EA artefacts 
[11, 12]. A discipline of evaluation in EA is needed because enterprises as well as EA practitioners in general 
require systematic, unbiased means of knowing if their products, practices, methods, and EA artefacts met 
their goals [11, 13, 14]. Evaluation leads to generate information that assist in making judgments and 
decisions program, service, policy, IS integration, and then to inform decision making to guide practical 
actions [11, 15-17]. 

EA evaluation enables the EA project’s stakeholders and enterprise architects to evaluate the 
developed EA artefacts and implemented ISs based on predetermined EA objectives [16]. Employing 
structured, usable, and effective EA evaluation model becomes vital in each EA project due to the outcome 
of the project. The common features which are provided by EA evaluation model are as follows: 
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. Investigating the integration of developed ISs 
. Investigating the IT and business alignment in developed EA artefacts 
. Assessing achievement of predetermined EA objectives 
. Assessing the performance of developed EA artefacts in response of EA objectives 
. Assessing the effectiveness of developed EA artefacts with future changes 
. Assessing the functionality of developed EA artefacts 

Aforementioned features of EA evaluation models and considering the importance of understanding 
the situation of implemented EA and achievement of EA objectives for project’s investors or stakeholders 
provide strong basis for the urgent need for effective EA evaluation model. The aim of this research is to 
further enhancing the development of effective EA evaluation model by initially identifying crucial issues or 
problems of existing EA evaluation models [18, 19]. 

There are some researches in the area of EA evaluation, however they did not consider post- 
implemented issues, and provide list of shortcomings. The goal of this research is to investigate the main 
shortcomings of the post-implemented EA evaluation models and approaches. 

The remainder of this paper is divided into the following parts:; the sresearch method of this 
research in described in Section 2; the current problems of existing EA evaluation models are presented in 
Section 3; the discussion and conclusion are given in Section 4, and 5 respectively. 
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2. RESEARCH METHOD 

This study employed the Systematic Literature Review (SLR) based on the provided guidelines that 
was proposed by Kitchenham and Charters (2007) in order to conduct the review [18] on finding issues of 
existing EA evaluation models. Hence, the main research question is: What are the main issues or problems 
in existing EA evaluation models? 

In total, 418 papers were identified and retrieved from 6 reliable databases, including: IEEE, ACM, 
Springer, Science Direct, Taylor and Francis, and Scholars. The Figure 1 and Figure 2 synthesize the result of 
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3. ISSUES EXISTING EA EVALUATION MODELS 
Main issues which are identified and summarized are explained as follows: 


3.1. Lack of structured EA Evaluation Model 

Most reviewed models are not based on any sound theoretical basis to provide the holistic 
understanding of EA evaluation and most do not provide structured mechanisms to evaluate the 
implemented EA. Moreover, most appraised evaluation models assume that the developed EA artefacts 
situations are well-defined with established business units and identifiable users who have well-defined roles 
and positions within the enterprise. The models are mainly developed based on the users’ experiences from 
previous projects and there is no theoretical foundation behind them. The EA evaluation models are 
frequently inapplicable in situations that may not have well-defined alignment between enterprise IT and 
business [11, 19, 20]. 


3.2. Lack of a guided method for EA evaluation 

Since the process of evaluation or analysis the EA artefacts within an enterprise is critical for the 
success of EA implementation, the process should be supported by an effective and useful method to 
facilitate the EA projects evaluation process. Most of the reviewed evaluation models do not provide the step 
by step guidance of the EA evaluation process. Employing an appropriate method for implementing the EA 
evaluation process will tremendously contribute to preventing users’ interpretation on the evaluation 
processes and activities [7, 21]. 


3.3. Complexities in EA evaluation models employment 

Most studied models do not provide easy to learn and simple practices for evaluating EA 
implementation. The typical usage of evaluation models in the EA implementation is to increase human 
understanding in complex matters such as the effectiveness of implemented EA artefacts and the achievement 
of defined EA objectives. In the context of EA implementation, stakeholders need to consider 
comprehensively the implemented artefacts of EA and competitive external environments to ensure a full 
understanding of enterprise situation. Existing EA evaluation models mostly present too detailed, which are 
time consuming and expensive practices, or too abstract which are not applicable in EA project. Besides, they 
introduce the main structures and concepts of evaluation in high perspective and do not provide holistic and 
appropriate metric for evaluation. Therefore, the evaluation faces many difficulties in terms of using the 
metrics, learning the practices, and implementing the practices [22-24]. 


3.4. Lack of a comprehensive EA evaluation model that covers all aspects of EA implementation 

Rapid changing enterprise requirements are the norm currently. Hence, there is a need to focus not 
only on the present requirements but also the requirements of the immediate future. This is not catered for in 
most of the reviewed models. The scope of requirements should not only focus on the main operational 
activities of the enterprise but should also cover the managerial activities [11, 25]. A wider scope of analysis 
that includes all the essential activities of the enterprise such as planning and coordinating, will enhance the 
enterprise planning capability in identifying an extensive variety of IS applications that can instigate and 
sustain the enterprise successfully. This has to be reflected in models for EA evaluation particularly in 
identifying and relating the requirements of the enterprise to its [Ss and information requirements. Existing 
models seem to stress on identifying cost and benefit of developed EA artefacts. Very few models continue to 
identify further information and data requirements for the EA evaluation which will be very beneficial to 
bridge the semantic gap from the business requirements to IT capabilities and developed EA 
artefacts [19, 26, 27]. 

The conclusion that can be made from the reviewed models is that they are mainly deficient in the 
ability to provide comprehensive useful functionality and make use of the developed EA artefacts effectively. 
Moreover, it lacks the ability to relate the business requirements to IT capabilities and specifically provide 
structured approach to facilitate further future changes analysis. Very few EA evaluation models offer 
comprehensive approach for evaluating all aspects of EA implementation. Table 1 shows the summary of the 
analysis made and highlights the analyzed features of each model. Subsequently, this research is directed 
towards addressing such deficiencies. It intends to develop an approach for EA evaluation approach for 
covering all aspects of implemented EA artefacts. The approach will be equipped with the mechanism to 
focus on the assessment of functionality and effectiveness of the EA artefacts. 
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Table 1. Summary of the Issues in the Studied Evaluation Models 








Not covering Lack of Difficulties | Lack of method 
Evaluation models all aspects of structured inEA for EA 
EA models evaluation evaluation 
F., & Tavana, M. (2012) [7] v v v 
Javanbakht, M., et al (2009) [28] v v 
Lange, M., & Mendling, J. (2011) [29] v v v 
Ojo, A., at al (2012) [30] v v 
Iacob, M. E., at al (2012) [31] v v 
Aier, S., & Schelp, J. (2010) [32] v v 
Van Steenbergen, M., & Brinkkemper, S. (2010) [33] v v 
Nakakawa, at al (2009) [34] v v 
Van Der Raadt, at al (2008) [35] v v v v 
Niemi, E., & Ylimiaki, T. (2008) [36] v 
Weiss, S., at al (2013) [8] v v 
Pruijt, L., at al [24] v 





4. DISCUSSION 

There are many existing evaluation methods and approaches in practice. Unfortunately, many 
models are not based on any theoretical foundations and lack good practices to accompany the proposed 
methods. These shortcomings lead to insufficient and inaccurate analysis of EA artefacts. Thus, the enterprise 
will not achieve the intended goal of the EA project. For more effective evaluation approach, comprehensive 
consideration should be given for requirement analysis in terms of functionality and effectiveness. 

Evaluation model should support the EA project lifecycle including the design, management 
(development), and maintenanceThere is also a lack of consideration regarding consistency between 
evaluation’s aspects in existing evaluation models. Consequently, the evaluation starts with using one 
approach, followed by other approaches that are not linked together. Besides, there is no specific practice for 
evaluating the functionality and effectiveness used by existing evaluation models. A usable evaluation model 
should focus on providing plan for evaluating the effectiveness and functionality of implemented EA 
artefacts. 

Monitoring and governing are the critical parts of EA implementation maintenance, and evaluation 
approach should provide appropriate plan for supporting these activities for each EA project. By doing so, the 
evaluation model assists architects and stakeholders to continuously improve the EA implementation and 
increase the quality of intended EA implementation goals. 

Effectiveness in EA refers to the ability of implemented EA on achievement of intended EA goals 
and provide appropriate integrated environment for developing EA artefacts. EA functionality describes how 
different components of enterprises, such as organizational units, business processes, people, and ISs, are 
related to each other and work as a whole towards the achievement of enterprise goals. Effective and usable 
EA evaluation method should consider effectiveness and functionality and use structured plan for evaluation. 
By doing this, effective EA evaluation method will enable to provide appropriate plans for governance and 
integration on developing accurate EA artefacts. 


5. CONCLUSION 

This paper has identified crucial issues of existing EA evaluation models. The investigation on EA 
evaluation literature carried out in this study provides a better insight of the current issues. By exploring the 
existing evaluation models, it was realized that there is no comprehensive and structured model and method 
in general for EA evaluation and specifically there is lack of focus towards assessing the functionality and the 
effectiveness of the EA artifacts. 

The findings of the study may provide impetus for further work in this area including the need for 
more post-implemented EA approaches. In addition, the findings emphasize the relevance of effective and 
usable evaluation method due to its impacts on an enterprise performance. With effective EA evaluation 
method will surely provide leverage to the enterprises as well as generate constructive values for them 
towards achieving their goals. The results of this research would be extremely beneficial for academics, 
practitioners, and enterprise’s decision makers in implementing and evaluating EA projects. 
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